The present invention relates to input and editing systems and particularly to manual data entry and gesture-based command systems employing input, editing and display functions.
A freeform graphics system is one in which a user can, by various means, input graphical data objects into a two-dimensional space and then edit the objects and their positions in the space. The system is xe2x80x9cfreeformxe2x80x9d in two senses. First, the graphical objects themselves may be of any shape, dependent only on the means provided by the input methods. Second, there are no constraints on where the graphical objects may be positioned in the space; the means for inputting and editing the objects allows the user to freely position and reposition objects. All of the graphical objects are independent of each other in that their relationship to each other is determined only by their placement in the 2-D space.
This second sense of freeform, which is also called xe2x80x9cunstructuredxe2x80x9d in this document, is the more important. A simple example of a freeform graphics system is a child""s xe2x80x9crubber stampxe2x80x9d graphics application, in which the user can use buttons (the rubber stamps) and a pointing device to input (xe2x80x9cstampxe2x80x9d) graphic objects anywhere on the drawing surface and then use the pointing device to move the objects around on the surface. An example of a non-freeform (xe2x80x9cstructuredxe2x80x9d) system is a simple text editor, in which all the characters are organized into an underlying xe2x80x9cstringxe2x80x9d structure.xe2x80x9d A character cannot be placed anywhere on the 2-D display, but only in positions where it can fit into the string structure (e.g. to move some characters to the right, some xe2x80x9cspacexe2x80x9d characters must be typed into the string to cause them to be pushed over).
A very useful type of freeform graphics system is a xe2x80x9cscribblingxe2x80x9d system in which the user can input freeform marks by simply drawing (xe2x80x9cinkingxe2x80x9d) them in a freehand manner. Any such mark entered is a freeform graphical object in the system; that is, the mark can be any shape the user chooses to draw it, and it can be drawn anywhere on the drawing surface. The input can be accomplished by a variety of means of sensing a freely movable input device, such as a stylus, a mouse, or even a finger (sensed by pressure or optical means), etc.
Editing and control operations can be performed on the graphical objects in a freeform graphics system by giving commands to the system. Conmmands can be given by conventional techniques, such as keyboard-based and menu-based techniques. But in a xe2x80x9cgesture-basedxe2x80x9d input system such as a pen-based notebook computer or large scale display known as a Liveboard (a trademark of Xerox Corporation), such control functions may be instituted by a command gesture. A command gesture is a handrawn stroke that is created with a stylus input device and interpreted by the system as designating an action for the system to perform.
A gesture-based system is capable of detecting that a user is entering a gesture command on a drawing surface instead of data usually by some action taken on the part of the user. Such an input action may entail holding or exerting pressure on a button on the stylus. The user instructs the system to execute the command by lifting the input device (e.g. stylus) from the drawing surface. Other methods are also possible. The system may recognize the input as an executable command, without any special user action, or may act in response to a double tap on the drawing surface, etc.
A common activity for users of gesture based computing systems, whether white-board-sized or notebook-sized, is informal scribbling in handwriting and sketches, whether for purposes of communicating with others or for making notes to oneself. The whiteboard system on the Liveboard is a prime example of a system to support this kind of activity. Scribbling systems treat this material as unstructured images or as collections of stroke objects.
A great deal of such scribbling activity involves generic structuresxe2x80x94lists, outlines, text, tables, and diagrams. To be effective, scribbling systems must support the creation and editing of these structures. Without such support, for example, the simple task of moving an item in a list can be tedious (move a segment of the list to make space for the item at the new location, move the item, close up the old space). This almost always takes too much time for users to actually perform.
Users of course could be provided with a set of explicit facilities for these generic structures. Thus, for example, the user declares that he/she wants to work with a table, and the system provides a tabular array which the user can fill out and manipulate. However, the problem is that users who are working in a xe2x80x9cscribbling mannerxe2x80x9d do not always know beforehand what structure they want. These structures are emergent. For example: users jot a couple of phrases; then they decide to make a more extensive list; then they begin to mark properties of a couple of items of the list; then they want to make these properties a second column (i.e., a table); then they want to make a sketch which violates any list or tabular convention; and so on.
Thus it is seen that the structuring is partial, both temporally and spatially. Sometimes the user wants to consider the material as a list, e.g., to re-order it, and other times he/she desires to consider it as a literal image and sketch on it. Furthermore, even if the user considers the material as a list, he/she may not want to regard the whole page as a list, e.g., just the left half of the page. It is desirable to have a system that can support this kind of fluid process. However, requiring users to explicitly declare structures often inhibits the natural flow of this kind of work.
There are scores of graphical systems supporting text, lists, outlines, tables, and diagrams. There are however very few that handle such input with a hand-held manually manipulable input device such as a stylus or pen. Usually, pen input is immediately recognized as characters, and then processed as conventional text. The commercial field of pen-based computing is very active and stroke-based outlining systems are beginning to appear. Such systems appear to be based on a notion of lined paper. The user declares a page to be a list, and then lines appear, creating a column of wide rectangular cells for the list items. See FIG. 3 for an example of one such system. Every stroke belongs to one of the cells 50, and in this way all strokes are grouped into list items. Every non-empty cell has a bullet 52 that represents the group of strokes in the cell 50. There are several gestures on the lines and bullets for manipulating the list structure, e.g., a line gesture from a bullet to a line moves the item, a pigtail gesture on a bullet deletes the item, and a gesture up or down from a line moves the line up or down (i.e., changes the size of the cell).
This system as well as other conventional systems however suffer from the inability to accept, modify or alter freeform information. Such information can take the form of text, graphics, tabular data and other forms of information all includable within the foregoing formats. Conventional data entry unfortunately cannot be entered in a freehand form without constraints such as defined borders or defined input areas. Editing operations can only act upon the entire information contained within the defined borders or input areas, not individual strokes within such borders.
It is desirable to have a system that would allow data of any type to be entered in a freeform manner without defined constraints such as borders or defined input areas or structures, not necessarily visible, e.g. text, and allow editing operations to be entered in a similar manner with editing operations being determined by the entered edit commands and with the constraints determining the nature of the data arising after entry of an editing command, rather than before.
It is an object of the present invention to overcome the disadvantages associated with the conventional systems described above.
The above object is achieved by a system which features a data input scheme wherein the structure of the entered data is irrelevant until the requirement is made to perform an operation with respect to that data. It is only when it becomes necessary to perform an operation with respect to the data that the determination of the structure of the data becomes important. The desired operation, which may include editing, rearranging, moving, etc., then results in a determination of the structure of the data in order to enable implementation of that specific operation. The structure of the data can be determined to be in the form of a list, table, or diagram. The entered data is freeform, meaning that its structure is not determined at entry. The data as entered can be free hand, scanned, graphic, textual, pictorial, typed, etc. It is treated by the system as freeform, meaning that it is treated merely as a series of strokes. Specifically, the scheme described above provides support for such forms of data in a flexible wayxe2x80x94both temporally (so that the user does not have to permanently xe2x80x9cdeclarexe2x80x9d structural modes) and spatially (so the user can delimit the extent of the structured material). The scheme is based on a set of design principles for specifying a set of structural/spatial models for any type of information on the display surface, a set of gestures for choosing particular structures and for operating on the information defined by such structures, and a set of system procedures for recognizing the structures and carrying out operations according to the structural models. These design principles are used to support handwritten lists (simple vertical lists), outlines (indented lists), text (within lists and outlines), tables (rows and columns), and diagrams (nodes and links).
In accordance with the present invention, a screen oriented display processing system is provided for the freeform entry of informational data including a display means, a.data entry means for entering informational data on said display means, a gesture entering means for entering freeform gestural data on said display, said gestural data representing a gesture associated with at least a portion of said informational data on said display, an assignment means responsive to the entry of said gesture for assigning an operation representative of said gesture to said portion of said informational data, and an implementing means responsive to said assignment means for implementing said operation on said portion of said informational data. The assignment means includes means for establishing a determination of structure of said informational data sufficient to accomplish said operation.
In accordance with another aspect of the present invention, a graphic editing system is provided including a data interface surface and user controllable means for generating information on said surface. The system comprises first means responsive to a user action that generates a line image on the surface for performing an operation on at least a portion of said information, said line image having a set of instructions associated therewith, second means responsive to said set of instructions for selectively analyzing, organizing, and rearranging information displayed on said surface, and third means for implementing said operation on said information according to said instructions.
In accordance with another aspect of the invention, a graphic editing system is provided including a data .interface surface and user controllable means for generating information on said surface. The system comprises first means responsive to a user action that generates a line image on said surface for selecting at least a portion of said information, said first line having a set of instructions associated therewith, second means responsive to said set of instructions for selectively analyzing, organizing and rearranging information on said surface, and third means responsive to said first user action for interpreting said information on said surface in accordance with said instructions.
In accordance with another aspect of the invention, a graphic editing system including a data interface surface and user controllable means for generating information on said surface. The system comprises first means responsive- to a first user action that generates a first line on said surface for selecting data thereon, second means responsive to said second user action subsequent to said first user action that generates a second line image having a set of instructions associated therewith, third means responsive to said instructions for analyzing, organizing and rearranging information on said surface associated therewith, and means for interpreting said information on said surface in accordance with said instructions.
In accordance with another aspect of the invention, a graphic editing system is provided including a data interface surface and user controllable means for generating information on said surface the system comprises first means responsive to a user action that generates a line image on the surface for performing an operation on at least a portion of said information, second means responsive to said line image generation for selectively determining a table, diagram, list, and outline model structures for said information on said surface in accordance with said operation, third means for identifying said information affected by said operation with respect to said structural model, third means for computing structural parameters for said operation as a function of objects identified, and fourth means for implementing said operation on said information.
In accordance with another aspect of the invention, a graphic editing method is provided for a system including a data interface surface and user controllable means for generating information on said surface. The method comprises the steps of drawing a line image on the surface in response to a user action for performing an operation on at least a portion of said information, selectively determining a table, diagram, list, and outline model structures in response to said line image generation for said information on said surface in accordance with said operation, identifying said information affected by said operation with respect to said structural model, computing structural parameters for said operation as a function of objects identified, and implementing said operation on said information.
In accordance with another aspect of the invention, a graphic editing system is provided including a data interface surface and user controllable means for generating information on said surface and memory means for storing information. The system comprises first means responsive to a user action for generating a line image on the surface, second means responsive to said line image generation for searching through memory and recognizing said line as a command gesture for performing an operation on at least a portion of said information, third means for selectively defining said information as a table, diagram, list, and outline model structures in accordance with said operation, fourth means for implementing said operation associated with said command gesture, and fifth means for displaying said information in accordance with said implementation.
In accordance with another aspect of the invention, a graphic editing method is provided for a system including a data interface surface and user controllable means for generating information on said surface and memory means for storing information. The method comprises the steps of generating a line image on said surface in response to a user action, searching through memory in response to said line generation and recognizing said line as a command gesture for performing an operation on at least a portion of said information, selectively defining said information as a table, diagram, list, and outline model structures in accordance with said operation, implementing said operation in accordance with said command gesture, and displaying said information in accordance with said implementation.
In accordance with another aspect of the invention, a graphic editing system is provided including a data interface surface and user controllable means for generating information on said surface. The system comprises first means responsive to a user action for generating a line image on said surface, second means responsive to said line image generation that recognizes said line as a command for performing an operation on at least a portion of said information, third means responsive to command recognition for defining said information on said surface as a list comprising at least one vertical column of line items separated by spaces in accordance with said operation, and fourth means for implementing said operation associated with said command gesture.
In accordance with another aspect of the invention, a graphic editing system is provided including a data interface surface and user controllable means for generating information on said surface. The system comprises first means responsive to a user action for generating a line image on said surface, second means responsive to.said line image generation that recognizes said line as a command for performing an operation on at least a portion of said information, third means responsive to command recognition for defining said information on said surface as a list comprising a plurality of line items, each having specific indentation level in accordance with said operation, and fourth means for implementing said operation associated with said command gesture.